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Introduction and summary 


In recent years policymakers have seized on teacher evaluation as a primary lever 
for improving schools. Of all school factors — from expanded school calendars 
to smaller class sizes to community and family engagement programs — teachers 
contribute the most to student achievement. 1 Policymakers reason that evaluating 
teachers based on their students’ performance will lead to the removal of under- 
performing educators and an improvement in the overall quality of the teacher 
workforce. Furthermore, there is growing evidence that effective evaluation sys- 
tems may prompt all educators to improve. 2 

In the past few years, nearly all states have passed legislation revising teacher eval- 
uation. Through Race to the Top, Teacher Incentive Fund grants, and No Child 
Left Behind waiver requirements, the federal government has pushed states to 
use student achievement as a major component in teacher evaluations. More than 
40 states and Washington, D.C., have responded by changing their laws related to 
education policy. 3 Teacher evaluation reforms and, in particular, efforts to assess 
teachers on the basis of student achievement have sometimes resulted in confron- 
tations between teachers and school districts. Chicago’s 2012 teachers’ strike is 
only the most recent example of bitter standoffs related to teacher evaluation. 4 

Yet confrontation and conflict are not the dominant themes in all districts seek- 
ing to reform teacher evaluation. Some districts have prioritized a collaborative 
approach to developing new evaluation systems and can provide valuable lessons 
for others to follow. But what exactly can be learned from such places? 

This report examines one district’s efforts to develop and implement a new 
teacher evaluation system in a cooperative manner with its teachers. For reasons 
of confidentiality, we refer to this district as the Studyville School District, a 
medium-sized school system in a northeastern urban center with roughly 20,000 
students and 1,600 teachers. It has been recognized at both the national and state 
level for the collaboration that has characterized the development and implemen- 
tation of the district’s new teacher evaluation program and for the use of student 
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achievement as a benchmark in its the evaluation system. 5 Leaders from the teach- 
ers union and district have worked together to adopt a framework for the new 
evaluation system through the collective bargaining process. They’ve designed 
and implemented the new system through joint labor-management committees 
and are working collaboratively to oversee the program. Teachers and school- 
based administrators have been central to the reform’s design and implementation 
throughout the process. 

To date, Studyville’s evaluation program has garnered support from key stakehold- 
ers, including teachers, school and district leaders, the teachers union, and the city 
school board. A key element of the reform is that it has real consequences: Some 
teachers have been recognized as exemplary while others have left the district 
because of poor performance. Furthermore, teachers report that the reform 
focuses their efforts more directly on student performance. 6 While the evaluation 
program’s direct effects on teachers’ instruction and student achievement have not 
yet been examined, stakeholders’ favorable views of the program thus far suggest 
that the reform maybe gaining traction in the district and that it could improve 
these key outcomes. 

In this paper we present an in-depth case study charting the evaluation program’s 
progress — from concept to initial design to implementation to the program’s current 
state. We examine aspects of policy and practice that have facilitated or impeded the 
program’s success and pay particular attention to developments in labor-manage- 
ment relations at the school and district level affecting the reform’s development. 

Based on our interviews with key district and union leaders and with a cross-section 
of teachers and school leaders, we are able to make the following observations: 

* Economic, political, and policy factors have facilitated the teacher evalua- 
tion program's development and acceptance. These factors help to explain the 
reform’s favorable review among many stakeholders, including teachers, school- 
based administrators, and district and union leaders. 

* Collaboration has been at the heart of the teacher evaluation program's 
creation and development. The program was conceived through the district’s 
2009 collective bargaining agreement with its teachers union. District leaders 
and union leaders worked together to create the evaluation program’s framework 
and oversee its implementation. Teachers and school administrators also col- 
laborated to develop many of the key components of the reform. 
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* The teacher evaluation program represents both a process and a product. 

The collective bargaining agreement laid out a framework for the teacher evalua- 
tion program and articulated a process through which the district would formalize 
the program details. This process led to a product, the evaluation system, which 
satisfied at least some of the needs of all constituents. Both the process and the 
product of the reform are important factors in the reforms progress to date. 

* The teacher evaluation program's progress reflects strong leadership 
coupled with broad input. Leaders at the city and district level were willing to 
work to bring the program to fruition, but they didn’t do it alone. They relied 
on teacher- and school-administrator working groups to develop many of the 
details of the program. The program therefore addressed the priorities of district 
and union leadership and reflected the needs of teachers and administrators. 

Some of what we learned from Studyville’s experience with its teacher evaluation 

program is summarized in the following recommendations: 

* Get out in front of a wave of reform, but be authentic to district needs. 

Our study district has made strides with its teacher evaluation program in part 
because it seized on momentum to improve teacher evaluation and staked 
out its ground with a collaborative approach to this area of reform. The school 
district’s status as an early implementer led to recognition from prominent 
government and labor leaders, which likely reinforced the reform’s value to key 
local stakeholders and encouraged them to continue to search for productive 
compromise. Moreover, by getting out in from of teacher evaluation reform, the 
district was able to set its own course for change and design its own system that 
responded to the district’s context and needs. 

* Invest in collaboration, but understand its challenges. Collaboration played 
a major role in the progress of this reform. While collaboration was important 
to the acceptance of the evaluation program, it is not the only ingredient driving 
reform progress. In the case of the Studyville School District, the context shaped 
collaboration among the various stakeholders, encouraging them to compro- 
mise and find mutually beneficial solutions. The lesson here is that investing in 
collaboration can lead to a better outcome for all, but this degree of collabora- 
tion is hard earned and depends on both the context and the people involved. 
Leaders of both management and labor should take stock of these factors when 
seeking to engage in collaborative reform. 
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• Pay attention to process and product. The evaluation program’s progress to 
date is as much due to the process through which it was developed as it is to the 
structural elements that comprise the reform. As a process, the program was 
developed collaboratively and over time. While the framework for the evalu- 
ation program was laid out in the collective bargaining agreement, the details 
were developed through joint labor-management committees. This process 
cultivated trust and understanding among parties and has been critical to the 
teacher evaluation program gaining traction in the district. This is even more 
important given that the evaluation program seeks to advance a common vision 
of effective teaching. The more time teachers and school leaders spend discuss- 
ing this vision in a structured way, the more likely it is to affect instruction both 
within and outside of the boundaries of the teacher evaluation program. 
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The study setting and the 
evaluation program 


Our study school district; which we refer to as the Studyville School District; is a 
medium-sized district in a northeastern urban center. More than 70 percent of the 
district’s approximately 20,000 students receive free or reduced-price lunches, and 
the district performs in the bottom 10 percent of districts on state achievement 
tests. The teacher evaluation program seeks to increase the instructional quality of 
its approximately 1,600 teachers. 

Implemented in the fall of 2010, the evaluation program requires that school lead- 
ers evaluate teachers annually and provide more frequent, informal coaching to 
improve instructional practice. These evaluations are based on the following three 
components: 

• Student progress towards specific performance-growth goals 

• Standards-based observations that follow the district’s teacher-evaluation rubric 

• Professional conduct 

Based on these three components, the evaluator provides a summative rating 
for teachers: 1 (needs improvement), 2 (developing), 3 (effective), 4 (strong), 
or 5 (exemplary). 

The system requires evaluators to observe and conference with teachers regularly 
throughout the school year. The evaluator uses standardized rubrics to evaluate a 
teacher’s instructional practice and his or her professional values. Additionally, the 
evaluator is expected to visit classrooms on a regular basis and provide ongoing 
informal feedback to teachers. 

Student performance comprises approximately 50 percent of a teacher’s evalua- 
tion score, with the exact weight varying based on teachers’ scores on the other 
components of the evaluation program. Each fall a teacher meets with his or her 
evaluator to set at least two performance goals (called student-learning objec- 
tives) for the academic year. Each goal is based on student growth in key skills 


The study setting and the evaluation program | www.americanprogress.org 5 


and knowledge related to the subject or grade taught. Elementary teachers in 
the fourth through eighth grades must create at least one goal based on the state 
standardized test. Teachers and evaluators again meet at the midpoint of the year 
to discuss teachers’ progress toward meeting their goals. Goals maybe modified 
slightly at that time if both parties agree to the change. 

Below we present typical goals that teachers might adopt as part of the evalua- 
tion program. 


Teacher goal setting 

Example 1 : High school media arts teacher. In arts and technology classes, where 
state standardized tests do not exist, students often develop portfolios of work 
showcasing their mastery of various skills. A media arts teacher might therefore set 
one goal of having 90 percent of his or her students complete a portfolio. For the 
second goal, the teacher might use the district's 21 st century skills standards and as- 
sessments to establish that 90 percent of students should advance one performance 
level from their fall baseline score on the rubric of essential skills in the strand of 
communication and collaboration by the end of the school year. 

Example 2: Elementary teacher. In elementary grades that administer the state test, 
it is possible to set performance goals that attempt to measure growth on the test. 
But since state test results aren't published until July, many teachers use district as- 
sessments, which are aligned with the state test, to set their goals. In the fall teachers 
in tested grades administer pretests and set goals based on those results. A teacher 
could, for example, set a goal of moving students' average score on the state English 
language arts standardized test from 50 percent to 70 percent "correct" over the 
course of the academic year. 


Based on evidence that they gather in the fall, district evaluators are required to 
identify potential exemplary and underperforming teachers by early November of 
the school year. These identifications trigger independent observations by third- 
party evaluators who are external to the district. These outside observers evaluate 
teachers concurrently but independently of school leaders, thereby verifying the 
leader s assessment of a teacher. This serves to “validate” teacher ratings. If both 
the school leader and the external evaluator rate a teacher as 5 (exemplary) on the 
instructional practice segment of his or her evaluation, and if the other portions 
of the evaluation (student growth and professional conduct) are sufficiently high, 
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the teacher earns an overall exemplary for a summative rating. These highly rated 
teachers become eligible for leadership positions. On the other hand, teachers 
scored as needs improvement, earning a rating of 1 from both internal and exter- 
nal evaluators, receive intensive support. If they remain in the needs improvement 
category by year’s end based on the three components of their evaluation, these 
individuals can be dismissed. 

The new teacher evaluation program, therefore, departs from the district’s previ- 
ous system in its mandate that teachers be evaluated annually, its effort to increase 
the quality and frequency of feedback (both formal and informal), the incorpora- 
tion of student achievement as a benchmark, and its clear positive and negative 
consequences for teachers based on their performance. 


Study methodology 

This paper presents a case study describing the teacher evaluation program’s devel- 
opment and implementation, which was guided by the following questions: 

• How did the district’s teacher evaluation program come to be? 

• What has facilitated the program’s development? 

• What has impeded its success? 


Data source and study sample 

To gain insight on the teacher evaluation program’s development and implemen- 
tation, we conducted in-depth, semi-structured interviews with a sample of 95 
participants: 10 principals, 10 assistant principals, and 72 teachers, including four 
teachers who were union stewards for their schools. We also interviewed three 
district-level leaders: the director of the teacher evaluation program, the president 
of the teachers union, and the assistant superintendent. Finally, we interviewed 
teachers and administrators at a purposive sample of 10 schools. Half were schools 
where teachers reported the most positive assessment of the evaluation program 
in district surveys conducted in 201 1, and half were schools where in the same 
surveys teachers, on average, reported the most negative assessment of the evalu- 
ation program. Four sample schools were high schools and six were kindergarten 
through eighth-grade schools. At each school, we interviewed the principal, assis- 
tant principal, and 20 percent to 25 percent of the school’s teachers. We selected 
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the teacher sample based on years of teaching experience and by subject and grade 
level taught. This allowed us to maximize teacher-rating variation. 

We interviewed participants during the 2011-12 school year, the second year of 
the district’s teacher evaluation program. We also conducted interviews at the 
start of the 2012-13 school year. Interviews were 45 minutes to 60 minutes long; 
they were audio-recorded and then transcribed. We asked participants to describe 
the origins of the evaluation program and that factors that they thought facili- 
tated or impeded its success. We also asked specific questions about the role the 
teachers union played in the creation and implementation of the program. Lastly, 
we inquired into the role of teachers and administrators (at both the district and 
school level) in developing and implementing the evaluation program. 7 
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Origins of the reform 


Although Studyville’s teacher evaluation program was implemented in the fall of 
2010, and the collective bargaining agreement that laid the groundwork for the 
reform was inked in the fall of 2009, the story of the evaluation program begins 
much earlier. Studyville had been building a foundation for evaluation reform and 
labor-management collaboration for more than a decade. 


An orientation toward collaboration 

The district’s teacher evaluation program grew out of a larger collaborative climate 
among key stakeholders, including the city’s mayor, the superintendent, and the 
president of the teachers union. The mayor, who came to office in the early 1990s, 
has administrative authority over the school district. In addition to being a member 
of the school board, the mayor appoints the board’s seven other members. In the 
past decade the mayor has made education a primary focus of his leadership. He 
secured funds for a major school reconstruction project, an effort that has led to all 
of the city’s schools being newly constructed or renovated during the past 10 years. 

The superintendent of the school district assumed his post a few years before 
the mayor came to office, so he was on the job for nearly two decades when the 
teacher evaluation program was adopted. During his tenure, the superintendent 
has overseen successful initiatives to increase the involvement of parents and care- 
givers in the education of their children. He also put an end to social promotion — 
the practice of promoting students to the next grade level based on their age rather 
than their demonstrated skills — in the district. 

Unlike his long-tenured counterparts, the president of the district’s teachers 
union, a longtime educator in the system who worked as a teacher and an instruc- 
tional coach for 28 years, came to office in 2007. The president was elected in a 
tight vote, after losing a close race for the union presidency in 200 1 . He promised 
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when elected to take a more “activist” role, enforcing the teachers’ contract more 
strictly and focusing attention on supporting less-experienced teachers. 

These three leaders cultivated personal relationships with each other over time. 
One union steward, commenting on the relationship between the three leaders, 
said, “They actually get along fairly well, and they argue but they seem to always 
come to some kind of commonality, which is pretty good. Otherwise nothing 
would happen.” 

This relationship bore fruit during the 2008-09 school year, when the district and 
union began to negotiate a new collective bargaining agreement. In the end both 
sides agreed on not only “bread-and-butter” issues such as salaries and benefits, 
but also on a major school-reform initiative that, along with the new teacher 
evaluation program, included an effort to increase college enrollment rates for the 
district’s students and a provision of differentiated levels of autonomy for schools 
based on their performance. The union steward credited the success of the nego- 
tiations to the relationship between these three men in these negotiations, stating 
that the contract’s success “is very dependent on the people. It’s very dependent 
on the union guys, the top-level administration guys who are on the [school] 
board, a superintendent, and his couple of underlings really being together for an 
extended period of time.” 


Why focus on teacher evaluation? 

As contract negotiations commenced, leaders on both sides agreed to reform 
teacher evaluation as one of three focal points in the effort to improve student 
performance in the district. They made this decision because all parties — the 
district, school leaders, and teachers — realized the evaluation system in place at 
the time wasn’t working and needed to be improved. The director of the district’s 
new teacher evaluation program recounted that under the prior evaluation system, 
“tenured teachers only had to be observed once every five years.” He added that 
it was “unheard of to terminate a tenured teacher for incompetence.” The union 
president agreed the old system was broken, saying that teacher evaluation had 
been “mixed bag.” He explained that “some principals were very thorough” but 
that many of the classroom observations were no more than “drive-bys” and 
“superficial at best.” He pointed out that teachers “wanted reform of evaluation,” 
seeing it as an opportunity to “grow professionally.” To sum up he said, “We all 
recognized this needed to be done. Teacher evaluation was broken.” 
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Widely held frustration with the teacher evaluation process in the district was 
brought into high relief with the hiring of an external firm to survey educators 
about their experiences with teacher evaluation. The results were overwhelmingly 
negative. Neither teachers nor principals thought teacher evaluation was work- 
ing. At the same time, publications such as “The Widget Effect” were advancing 
teacher evaluation as a key lever for school reform. 8 These factors help to solidify 
the district’s decision to focus on improving teacher evaluation as a vehicle for 
improving student achievement. 


Roots in collective bargaining 

Although the district could have created a new evaluation system with little union 
involvement, district leaders made a deliberate choice to set down the roots for 
the new teacher evaluation program in the 2009 collective bargaining agreement 
between the school district and the teachers union. The contract with teachers 
and the negotiations that produced it were significant for two reasons. First, it 
visibly signaled a stronger partnership between labor and management than had 
previously existed in the district, reflecting a shift toward a more collaborative 
and experimental approach to reforming schools on the part of the city, school 
leaders, and teachers. Second, in negotiating the teachers’ contract, district and 
union officials focused on the principles underlying teacher evaluation reform and 
embedded these in the final contract. The details of the district’s eventual teacher 
evaluation program were resolved outside of the formal contract negotiations. 

This approach prevented the program’s development from becoming bogged 
down by unrelated concerns and reduced the possibility of the evaluation pro- 
gram being held hostage to bread-and-butter issues such as wages and benefits. 

While the contract received largely positive reviews as an important step forward 
in teacher evaluation reform, several news editorials argued that the reform was 
not as dramatic as it at first seemed to be because it was not embedded in the 
terms of the teachers contract. The president of the teachers union said some 
roundly criticized the final contract, asking, ‘“But where’s the TEP [teacher evalu- 
ation program] ?’” The assistant superintendent said others disparaged the con- 
tract, calling it “an agreement to agree.” 

District and union leaders, though, believed that this perceived weakness was 
actually a key strength. The union president argued that attempting to hammer 
out the details of evaluation reform during negotiations would have been too 
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cumbersome. Instead it was decided to root the teacher evaluation program in the 
contract through a side-letter agreement. This agreement was developed concurrent 
to the negotiations over wages and working conditions, but, as the assistant super- 
intendent described, “at an entirely separate table” and in a “separate room.” This 
physical and symbolic division of “bread-and-butter” issues from reform concerns 
was “important,” according to the assistant superintendent. Relatively free from the 
strictures that govern collection bargaining, representatives, at what was termed 
the “reform table,” were able to have, in the words of the assistant superintendent, 

“a freewheeling discussion of reform issues.” In addition to district and local union 
leaders, state-level union leaders, representatives from the mayor’s office, and lawyers 
with expertise in labor relations were seated at the “reform table.” 

Shortly after the conclusion of contract negotiations, the president of the teachers 
union faced the daunting challenge of securing agreement from his membership. 
According to the union president, it was clear in discussions leading up to the 
vote on the proposed contract that some teachers were skeptical about the idea of 
teacher evaluation. Several union stewards mentioned that some teachers thought 
it was what they termed a “salary dump” — a convenient and easy way to get rid of 
higher-paid veteran teachers and reduce the district budget during tight economic 
times. As one union steward observed, “It’s just very scary, especially with the 
economy the way it is. I mean, teachers who have been here for a while pretty eas- 
ily could be replaced by three teachers who would cost the district as much money 
as that one longstanding, tenured teacher.” The union president admitted that his 
membership also had concerns about being fired if their “students did lousy on the 
test.” Eventually, union leaders convinced the skeptics and calmed fears, winning 
membership support for the contract and the new teacher evaluation program. 

The union president credited his memberships’ buy-in to the fact that “teachers 
felt more confident because they had input.” 

Had the teachers rejected the proposed contract, the district and union would 
have entered into a binding arbitration process to resolve the contract — an 
outcome neither side wanted. An external arbitrator would have reviewed the 
various demands and made a binding judgment regarding how negotiations would 
be settled. This threat of binding arbitration pushed teachers and administrators 
toward accepting the contract, as union and district leaders felt that any alterna- 
tive imposed by an external arbitrator would have been worse. Ultimately, teachers 
voted by an overwhelming margin of approximately 22 to 1 to ratify the contract, 
which gave them what they sought — the maintenance of their health benefits and 
annual pay raises over the four-year life of the contract. 
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Administrators also received much of what they wanted — a commitment to the 
teacher evaluation program and an agreement that the process for creating the 
evaluation system would be developed in a joint labor-management commit- 
tee and spelled out in a side-letter agreement to be appended to the contract. 

The details of the teacher evaluation program would subsequently be developed 
through committees of teachers and administrators. The president of the teachers 
union stated that the inclusion of the teacher evaluation program in the contract 
via a side-letter agreement was significant: “This time it [the evaluation program] 
was in the contract; whereas other reforms weren’t.” He said this meant that the 
district “couldn’t walk away” from the program. 


Facilitating factors 

The teacher evaluation program, and the contract that created it, represented a 
significant change for the school district. It required leaps of faith from teachers, 
school administrators, and district and union leaders. Not only did it establish 
a whole new way of evaluating teachers and leaders, but it also advanced a new 
process for reform through collaboration. If the district or union had backed out, 
or if teachers or principals had lost faith in the effort, the reform effort could have 
folded. What enabled the program to grow and move steadily forward? 

Studyville’s investment in the teacher evaluation program was facilitated by a 
number of things, most significantly the economic context, the policy context, 
and the political context. One union steward argued that these three factors, along 
with the contract negotiations themselves, spurred the district to create the evalu- 
ation program: 

The mayor wanted to do something different. The union contract was coming. 

The president [President Barack Obama] had it on his table with education , 
and the economy was an issue at the same time. ... those four things kind of came 
together to allow [Studyville] to really put this together and jump out of the box. 
And if it was not a union negotiating year, I don't think we'd have seen the same 
results. If it was not a bad economy, we might not have the same results. 
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The economic context 


Study participants agreed that the economic context — the prevailing economic 
winds of the late 2000s — played a large role in the evaluation program’s adoption. 
The stock market plunged in 2008 and bottomed out in March 2009. 9 The local 
economy was struggling as well. The mayor was projecting large budget shortfalls 
and warned that layoffs might occur. A union steward explained the economy’s 
influence on the teachers’ willingness to support the evaluation program. 

"We were afraid to lose our jobs. In other towns, [teachers unions] were going out 
and demanding arbitration, losing arbitration ... and not voting for their new 
contracts. ...as bad as the economy is right now [fall of 2011], it was much 
scarier two years ago when all this was going on. The banks were failing. Most 
people were saying: Were entering a great depression.' So I think most people 
in the union were like — ‘I'm working. I’m going to work today.’ When we took 
the vote [on the contract], there were people that dissented but ... it was a very 
small minority." 

The union president echoed this steward’s sentiment, suggesting that the economy 
forced teachers and administrators toward compromise, which played out in the 
contract negotiations and later in support of the reform. 


The policy context 

The policy context — the landscape of education reform in 2009 — gave Studyville 
stakeholders reasons to collaborate on teacher evaluation reform. Nationally, 
policymakers were focused on reforming teacher evaluation systems that were 
widely perceived as broken. In July 2009 President Obama launched the Race to 
the Top initiative, which included teacher evaluation as a major focus of reform 
and provided incentives to states to permit teachers to be evaluated based on 
student test scores. Moreover, it encouraged districts and unions to work together 
to improve schools. The mayor, superintendent, and president of the teachers 
union recognized that their proposed teacher evaluation program fit several of the 
specifications of Race to the Top. The mayor wanted the school district to adopt 
a collaborative teacher evaluation reform that would make it more competitive in 
seeking federal education grants, including Race to the Top. In fact, when the state 
applied for Race to the Top later in 2009, it included the district’s new contract 
with its teachers in its application. 
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The political context 


Lastly, the political context made Studyville ripe for reform. First, the mayor was up 
for re-election and wanted to make education a focal point of his campaign. Being 
involved in a high-profile struggle with the teachers union could have put a damper 
on his re-election bid. Second, the mayor wanted the school district to be seen as a 
model of collaborative school reform. As a result, he was willing to compromise to 
ensure that reform happened and that the process went relatively smoothly. 

Furthermore, there was growing rhetoric on the national stage that was critical of 
teachers — and of their unions in particular. As one building principal noted: 

[T]he message has been very clear you know, from Washington and throughout 
the country: “Teacher, you are the problem and we are going to face it. Were 
going to hold you accountable to the nth degree for student learning. We don’t 
want to hear about excuses ... and we have the instruments to be able to mea- 
sure that [student learning].” 

Just like in Studyville, teachers and their unions across the country were looking 
for good press to combat negative portrayals in the media. Some district teachers 
and union leaders saw the teacher evaluation program as a way to demonstrate 
their openness to reform while also preserving a role for themselves in its design. 
This need also led to involvement from state and national union leaders, who pro- 
vided support and encouraged a collaborative outcome that could be a model for 
agreements across the country. For this reason, leaders from the state and national 
union affiliate were present in the negotiations that led to the development of 
Studyville side-letter agreement. 
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Reform: A process and a product 


With the teacher evaluation program ensconced in the contract via the side-letter 
agreement, the district and teachers union went about designing the details of the 
reform. They began by developing a set of common beliefs to guide discussions. 


A collaborative process 

The design process leading to the development of the district’s teacher evalua- 
tion program was explicitly collaborative and deliberate. In late 2009 the district 
formed three committees to carry out the district’s broader reform agenda: a 
citywide reform committee consisting of three teachers, three administrators, two 
parents, and the assistant superintendent (ex officio); a joint committee devoted 
to developing the teacher evaluation program composed of six teachers and six 
administrators; and a committee devoted to developing teacher and caregiver 
surveys composed of four teachers and four administrators. Teacher and adminis- 
trator working groups supported the work of these committees. 

The teacher evaluation program joint committee met on alternate weeks. During 
the “off weeks,” members of the joint committee met with working groups to 
develop specific pieces of the reform model and also to solicit input on ideas from 
the joint committee from their constituencies. This iterative process continued 
throughout the winter and spring of 2010, when the teacher evaluation program 
committee made recommendations to the school board regarding the program. 
The program’s development was therefore explicitly and deliberately collaborative. 
The director of the evaluation program observed, “Collaboration has been a huge 
part of our process and development of our process. I don’t think another district 
could just adopt the evaluation program. A large part of why it works is because of 
the conversations we’ve had along the way.” 
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Furthermore; the process ensured that teachers played a central role in the 
reform’s development. “Teachers had tremendous input,” said the program’s direc- 
tor. “Any teacher could participate in the development of the program,” she added. 
The union president described attending joint evaluation reform meetings every 
other week and meeting with a working group of teachers at the union office on 
the off weeks. Some 40 or 50 teachers would sometimes attend the meetings to 
hear updates from representatives of the working groups and to work on issues 
that arose in these sessions. 

Even beyond the involvement of individual teachers, the union was deeply 
engaged in the process. “The union was very heavily involved from the beginning 
and they continue to be,” said the evaluation program’s director. A principal stated, 
“I think they [the union] were great. I mean, there were union people on the com- 
mittee that I was on. ... it was a collaborative effort and I think everyone realized 
that — everyone realized that this is what’s right.” 

In addition to teachers, administrators also participated in working groups and 
had their input incorporated into the process. One principal who was a member of 
a working group emphasized the collaborative aspect of the evaluation program’s 
creation and implementation: “We met many, many, many times. And it was 
definitely a collaborative effort.” Discussing the teacher evaluation instrument, she 
added, “I’m very happy with it — just the ability to have your meetings that you are 
making the goals together. It’s not top down.” 


The benefit of an extended development timeline 

Participants in our study emphasized the benefits of time spent developing the 
teacher evaluation program. They said that taking the time to develop the sys- 
tem collaboratively had three main benefits: enhanced trust between labor and 
management, the opportunity to communicate the purpose and details of the 
evaluation program to important constituencies, and ultimately, a more robust 
evaluation system. 

The time spent developing the teacher evaluation program permitted the design- 
ers to consider a broader range of possible logistical glitches and unintended con- 
sequences, thus preventing them from arising. According to the union president, 
the investment of time on the front end led to a better system. “There were lots of 
work and meetings — tons of input. “We argued about it. We thought about all the 
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hypotheticals.” He noted that because of the initial effort, the actual implementa- 
tion of the program “went smoothly.” 

The time spent in the development stage also provided the opportunity for admin- 
istrators and teachers to gain a better understanding about what constitutes good 
instruction. The union president explained, “We knew we wanted to improve 
instruction but didn’t have a good handle on what good instruction was. . . . there 
were 48 principals and 48 different conceptions.” To develop a shared framework, 
he said, “principals and teachers worked side by side defining effective teaching.” 
This work drew on Charlotte Danielson’s “Framework for Teaching,” 10 as well as 
on the personal insights of teachers and administrators about effective teaching. 
Importantly, it also included a rubric. Although the district had previously used 
Danielson’s work, the inclusion of a rubric and subsequent discussions of what 
the rubric meant in terms of concrete practices made the district’s instructional 
expectations more explicit this time around. 

The time spent also allowed relationships to develop. “We spent one school year 
working on [the teacher evaluation program] collaboratively,” noted the union 
president. “That year spent is the best thing we did. It couldn’t be done in three 
months.” He went on to explain that because of the initial groundwork that went 
into developing the reform, its rollout and implementation has been less prob- 
lematic. “We never got completely stuck because of the preparation we did in year 
one.” As a result, when issues arose with the program, the process never broke 
down. Instead the union and district leaders have continued to communicate 
relatively openly to resolve any differences. 

Lastly, the time spent up front enabled leaders to communicate the purpose and 
details of the evaluation program to teachers and school administrators as they 
were worked out. This proved to be very important because when the system was 
implemented in the fall of 2010, some details were still being decided. “I spent a 
year messaging, keeping everyone apprised,” said the union president. This mes- 
saging continued in the program’s first year, the 2010-11 school year. One assistant 
principal recalled that her building leader relayed messaging constantly to ease 
teachers’ concerns. “The principal kept saying, ‘This is new. They’re introducing it. 
Don’t panic. It’s not meant to hurt anyone.’” 

She added that the union was communicating deliberately as well. “One of the 
things that our union kept doing, which I loved, was saying to the teachers — ‘We 
were part of this. We agreed to this. . . . this is something that was really thought 
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out. It’s not meant to hurt.’” Another principal offered a similar observation, say- 
ing that “real clear messaging from the union” at the beginning of the program’s 
second year served to “clarify any other misconceptions” teachers were holding 
onto from the first year. “I think we opened this year [the second year] crystal 
clear,” he said. 

While the new evaluation system inevitably encountered bumps in the road, the 
time that was spent developing the teacher reform led to a smoother implementa- 
tion. Clearly, none of the challenges that arose derailed the process or compro- 
mised the support from teachers and administrators, in large part because these 
groups felt that their voices had been heard in program design. They believed 
that the program had been designed with care, that all parties had forged strong 
relationships, and that the details of the system’s key elements had been clearly 
communicated to teachers and administrators. 


System design 

In addition to the collaborative process used in developing the teacher evaluation 
program, the district made six key decisions in designing the system that facili- 
tated the reform’s acceptance and perceived success. 

First, leaders on both sides decided to emphasize the reform’s potential to affect 
teachers’ practice through coaching rather than its potential as a mechanism to 
dismiss underperforming teachers. Both sides recognized, however, that some 
underperforming teachers needed to leave the classroom, and the evaluation 
system provided for that outcome. The district wanted to recognize exceptional 
teachers, but the bulk of the district’s attention has remained on improving the 
instructional practices of teachers. In the words of the evaluation program’s direc- 
tor, “The most powerful aspect of the program is the conferences and conversa- 
tions [between teachers and administrators], not the ratings [of the teachers].” 

Although this comment could be construed as proverbial “spin” to make the new 
system more palatable to the union membership, the district’s actions actually 
lend further credence to its authenticity. The district has not focused on evaluating 
the rigor of goals, but has instead invested in the coaching relationship between 
teachers and administrators. As the evaluation program’s director noted, “We 
haven’t checked goals [student-learning objectives] ... because in some ways that 
would detract from the teacher/ administrator relationship and ownership over 
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goals.” This decision, she said, was made deliberately in order to ensure that teach- 
ers and evaluators could set goals that were most likely to bring about instruc- 
tional improvement. 

A second key design decision was to incorporate student achievement into teacher 
evaluation by using student-learning objectives rather than value-added measures, 
which are based solely on external calculations using standardized test scores. This 
was a deliberate choice on the part of designers. The decision was based in part 
on practical concerns about value-added measures. The director of the evaluation 
program noted that all teachers can develop student-learning objectives, “but 
VAMs [value-added measures] can’t apply to everyone.” This is because many 
teachers don’t teach in subjects and grades that have adequate student testing. 
Additionally, the director noted that the real value of student-learning objectives 
comes from the fact that, unlike value-added measures, they are jointly developed 
between teachers and administrators, which gives teachers’ a sense of ownership. 
She said that the district’s teacher evaluation program “is not a hard measure of 
student achievement. Its strength is that teachers are more invested in their goals.” 
The assistant superintendent elaborated, saying, “In districts using value-added, 
there is a lot of discussion about validity and accuracy and consistency . . . [and] 
the primary goal is sorting teachers.” By contrast, he said, “our priority is the for- 
mation of that professional learning culture” based on “professional conversations 
and feedback loops.” He added, “we’re far from perfect,” but “we realized that it 
[the teacher evaluation program] didn’t have to be perfect to be good.” 

Although their measurement properties can be questioned, student-learning 
objectives support the designers’ focus on promoting conversations and self- 
reflection on the part of teachers. It is also worth noting that the use of student- 
learning objectives reflects the notion that teachers would be more motivated 
to improve their practice if they believed in the importance of their goals and 
understood what it would take to meet them, two areas in which districts using 
value-added measures have struggled . 11 

Third, the district’s teacher evaluation program required annual evaluations 
for teachers and administrators. Participants — both teachers and administra- 
tors — said that it was critical that teachers be evaluated more frequently than was 
the case with the earlier evaluation system. Teachers are now officially observed 
several times during the year, and evaluators are expected to conduct informal 
classroom visits and provide feedback even more regularly. Annual evaluations, 
though time-consuming to conduct, aligned with some of the policy recommen- 
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dations receiving vocal support in 2009. 12 Furthermore, they signaled to teachers 
and administrators the importance that the district placed on evaluation. 

Fourth, the school district instituted evaluations for school- and district-level admin- 
istrators that paralleled the teacher evaluation program. The district also created 
surveys for teachers and community members that factored into the evaluations of 
school administrators. This effort to promote “top to bottom accountability” was 
cited by administrators and teachers as a key aspect of Studyville’s larger reforms that 
facilitated the teacher evaluation program’s success. As one principal noted, “It’s not 
just the teacher. The principal is [also] being looked at. It’s what I’m being rated on 
as well, so it’s us. It’s not the administration [on one side] and the teachers [on the 
other]. It’s us together moving the children.” Administrators also said that the more 
comprehensive and inclusive evaluation system had the added benefit of giving 
school leaders the opportunity to model how to be evaluated, allowing them to cul- 
tivate trust and teamwork with their faculty. One principal said, “One of the things I 
did was when I self-evaluated for my evaluation, I shared it with them [teachers]. So 
they saw that I gave myself a couple of 2s [in areas] that I was developing on. So that 
really built trust once I did that.” 

A fifth key feature was the inclusion of an external validator in cases where teach- 
ers were identified as potentially receiving a 1 (needs improvement) rating or 5 
(exemplary) rating. This validation process increased teachers’ trust in the evalua- 
tion program. Providing another layer of teacher confidence is the fact that union 
leadership reviews the list of external evaluators and can remove anyone it does 
not want in the position. Administrators and validators observe the same lesson 
and rate it independently. For teachers in jeopardy of being rated 1, this validation 
must occur for three separate lessons. Teachers can only receive a 1 or 5 designa- 
tion if, at the end of the evaluation cycle, both the evaluator and external valida- 
tor give the teacher’s instruction the same rating and the other portions of the 
evaluation are sufficiently low or high, respectively. 13 Teachers felt the inclusion of 
the outside validator was critical, particularly in the case of low-rated teachers who 
are at risk of losing their jobs. “I think the greatest thing in the teacher evaluation 
program is the validator,” said one teacher who had initially received a rating of 1. 

Lastly, a key aspect of the design was the creation of rubrics to guide observation 
of instruction. The program rubric described effective instruction more specifi- 
cally than the district had ever described it before. Because observations were 
becoming more high-stakes, the school district had to take steps to standardize 
observations within and across observers. Relying on these instructional rubrics 
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made this work easier: Evaluators could point to specific practices rather than 
rely on broader judgments of a teacher’s performance. But the creation of rubrics 
also promoted ongoing learning and development on the part of teachers. As 
one principal said, “I definitely was a cheerleader when it came to the rubric that 
teachers were able to take to look at and say ‘Okay. Well, this is where I am and 
this is where I want to be.’” Participants reported that the rubrics allowed them to 
understand what was expected of teachers. 
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Implementing the teacher 
evaluation program 


The new teacher evaluation program represented a substantial change from the 
prior system. Despite the time spent developing it, implementation was not 
simple. It required school-level administrators, teachers, and district-level admin- 
istrators to adopt different behaviors related to evaluation, and the learning curve 
was steep for many of these educators. As a result, several challenges arose in the 
program’s first year. 

One major challenge related to the training of administrators to carry out the pro- 
gram. Like many evaluation reforms, Studyville’s program hinges on the will and 
skill of administrators and requires them to devote much more time to evaluation 
than many had been used to devoting. Not only does the program require admin- 
istrators to rate teachers using a standards-based rubric, it also requires more con- 
ferences with teachers than the previous evaluation system mandated. Moreover, 
the district’s new teacher evaluation program relies on administrators working 
with teachers to set “rigorous and reasonable” student-learning objectives. Both 
teachers and administrators were uncertain about what “rigorous and reasonable” 
meant or how to operationalize it going into the program. 

It was a challenge to provide sufficient training to administrators to allow them to 
implement the program fully in 20 1 0, its first year. “No one had been trained by 
July,” recalled the union president. Summer trainings were held, but “training was 
limited and compressed,” according to the president, and some administrators 
could not attend the sessions. Administrators and teachers reported that admin- 
istrators did not have a consistent understanding of the process or specifics of the 
evaluation program when it launched in the fall of 2010. As a result, the union 
president observed, “lots of administrators weren’t implementing it correctly in 
the first year. They defaulted back to their old practices.” 

The district recognized this inconsistency as problematic and hired a full-time 
director of teacher evaluation early in the 2010-11 school year. Since then the 
director has worked to increase the standardization of practices across evaluators, 
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ensured that trainings happened, communicated about the system to teachers and 
evaluators, and provided support to those implementing the system. The person 
appointed to the position had served as a longtime teacher and administrator in 
one of the city’s high schools and, according to participants, was well-regarded by 
all stakeholders. This helped her achieve credibility from the start and be accepted 
as a fair manager by all sides. 

A second challenge concerned data management. With administrators rating 
teachers’ instruction at regular intervals, setting and revising student-learning 
objectives with teachers throughout the year, and considering other forms of 
data simultaneously, along with keeping track of the data, became a major con- 
cern. In the program’s first year there was no good system in place. Case in point: 
Administrators routinely emailed Microsoft Word documents with key details and 
teacher ratings back and forth to central office. This process was inefficient and 
unnecessarily time-consuming for evaluators, and it also raised privacy concerns. 

According to one principal, the teacher evaluation program’s “minutia was dif- 
ficult.” Her assistant principal provided the example of the difficulty of tracking 
program data in the first year: “ [Teachers] would type something on their teacher 
evaluation [document] then I’d have to get it on mine or we would have to replace 
mine. At the end they were giving me their data and I was actually typing it in for 
them.” She added that the process was stressful for teachers and that she offered 
to enter their data because, as she told them, “This should not make you crazy.” 

She said everyone quickly realized that this was not an effective or efficient way to 
manage data. In the program’s second year, the district put a new online system in 
place to track educator performance. 

A third challenge related to the limitations of the evaluation program from the 
perspective of school administrators. Despite efforts from district and reform 
program leaders to frame the evaluation system in terms of teacher develop- 
ment, some school administrators perceived the system as an opportunity to 
remove persistently under-performing teachers without following the necessary 
district policies and procedures. Like the previous system, however, the teacher 
evaluation program has an established process that administrators must follow 
and safeguards to protect teachers from unwarranted efforts to remove them. 
One union steward said: 
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A lot of the principals came right out of the box labeling [teachers] Is on previ- 
ous stuff, not on what was going to happen in the teacher evaluation program. It 
was kind of was like housecleaning. Our particular principal came in and said 
[to one teacher], 'You’re a 1.’ She didn’t even read the goddamn teacher evalua- 
tion program. ... she just didn’t like the teacher out of the box. And so she [the 
teacher] wasn't a team player within the first month, so automatically she wasn’t 
scored well on her evaluation and from then on, it was game on. So as a union 
rep, I spend a lot of time defending that position and trying to implement the 
evaluation program correctly. 

The district’s director of the evaluation program voiced a similar concern, albeit 
less forcefully. She reported that some administrators have been disappointed 
that teachers they perceived as underperforming ultimately remained in district 
schools. For the lowest- and highest-performing teachers, student performance 
counts for more than 50 percent of the final rating. As a result, the administrator’s 
assessment of instruction does not represent the teacher’s final evaluation score. 
This practice limits the impact of administrators’ subjective views of teachers’ 
practice. The director explained the effect of this, saying, “For a lot of teachers [in 
danger of dismissal], quite honestly, student-learning growth saves them.” 
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Outcomes of the teacher 
evaluation program 


The purposes of the evaluation system are to raise student achievement in the dis- 
trict through several mechanisms: by providing feedback to all teachers so they may 
improve ; by recognizing excellent educators, and by providing a clear, efficient, and 
fair method for removing teachers who consistently underperform. Studying the 
overall effect of the teacher evaluation program on student achievement is challeng- 
ing because the district adopted several reforms at the same time, which makes it 
hard to isolate the contribution of the evaluation program. There is strong evidence, 
however, that the evaluation program affected each of these mechanisms. 

The clearest outcomes of the program are in the number of teachers who have 
been validated as 5 (exemplary) and the number of teachers who have left the dis- 
trict for performance reasons. In Table 1 below, we present these outcomes during 
the program’s first two years. As seen in the top panel, 36 teachers were validated 
as exemplary during the 2010-11 school year, and 66 teachers were validated as 
exemplary the following school year. Interestingly, evaluators initially identified 
many more teachers as exemplary in 201 1-12 than in 2010-11, but a smaller pro- 
portion of these teachers were validated as exemplary by the end of the year. 
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TABLE 1 


Teacher outcomes in the teacher evaluation program 



2010-11 

2011-12 

Total number of teachers evaluated 

1846 

1457 

Exemplary teachers 

• Initially identified as exemplary (5) 

40 

95 

• Validated as exemplary (5) 

36 

66 

Underperforming teachers 

• Initially identified as needs improvement (1) 

75 

58 

• Validated as needs improvement (1) 

38 

19 

• Validated as developing (2) or effective (3) 

29 

20 

• Resigned mid-year 

8 

19 

Resignations due to performance 

• Total resignations 

34 

28 

• Tenured teachers 

16 

17 

• Nontenured teachers 

18 

11 


It’s not surprising that the teacher evaluation program has led to a substantial 
uptick in teacher departures for performance reasons. As seen in the second panel 
of Table 1, 75 teachers in 2010-1 1 and 58 teachers in 201 1-12 were identified as 
potential Is (needs improvement) at the beginning of the year. Some of these 
teachers improved and were validated at a higher performance level by year’s end. 
Others saw the proverbial writing on the wall and resigned from the district before 
the final evaluations were completed. Of those who were validated as 1 (needs 
improvement) at the end of the year, many were notified that they would be 
dismissed. A few teachers were given a second year to demonstrate improvement, 
and most of these did indeed improve their instructional practice. 

All of the educators who were notified that they would be dismissed ultimately 
left the district voluntarily. In total, district leaders reported that 34 teachers, 
including 16 tenured teachers, left the district because of their evaluation ratings 
in 2010-11. This represented approximately 1.3 percent of tenured teachers and 
2.8 percent of nontenured teachers in the district. Twenty-eight teachers (1.9 
percent) left the district after the 2011-12 school year. Four principals or assistant 
principals left in the program’s first year, and after the program’s second year, four 
administrators departed for performance reasons. The exit of tenured teachers 
for performance reasons is a marked departure from the district’s record under 
its previous evaluation system. As noted above, the evaluation program’s director 
said, “In terms of tenured teachers, it was probably unheard of that anybody was 
terminated for incompetence [prior to the new evaluation system].” 
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Interestingly, participants noted that the evaluation system has made it more chal- 
lenging to non-renew nontenured teachers. According to the director, prior to the 
program, “it was pretty easy to non-renew someone. You just basically had to meet 
the deadlines and have evaluated them.” The union president concurred. “Under the 
previous system nontenured teachers had no rights pre-April 1,” of their first year, 
which is when their legal protections increased. “Administrators could just tell them 
they were fired. . . . now administrators have less leeway to non-renew.” The direc- 
tor of the evaluation program said that tenure and evaluation “law hasn’t changed 
but we’re treating them the same as we’re treating tenured teachers.” As a result, she 
explained, nonrenewal “is not just automatic.” This has caused “frustration for princi- 
pals,” the director noted. But the process has become fairer for both nontenured and 
tenured teachers, and it is now aligned closely to the evaluation system. 

Largely because the evaluation process was clear, providing teachers with direct 
information about their performance, and because the union worked with the district 
to oversee the program, underperforming teachers chose to leave voluntarily. “We 
have not gone to termination hearings on anyone,” reported the evaluation program’s 
director. The district has instead encouraged teachers to sign separation agreements 
that offer departing teachers a few extra months of pay and avoid legal proceedings. 
The director explained that such agreements “allow people to resign with dignity and 
to really make a decision on their own in terms.” She added, “It’s better to pay some- 
body for a couple extra months than to fight litigation on a termination.” 

District and labor leaders have viewed these results differently. While asserting that 
the teacher evaluation program is “consequential” based on the number of teacher 
departures, the district continues to place more emphasis on the program’s capacity 
to improve instruction. District leaders regularly cite the number of teachers who 
were identified as potentially in the needs improvement category early in the school 
year but showed sufficient improvement to be rated higher by the end of the year. 

Union leaders have framed the number of teachers who have departed the district 
for performance reasons as a positive to their members. Teachers have been gener- 
ally supportive of these moves because they recognized that some teachers needed 
to be dismissed. They also saw that the administrators were not using the evalua- 
tion system capriciously to fire teachers. The union president said that teachers felt 
the number of teachers dismissed was lower than many feared. “At the end of year 
one the district didn’t fire 300 teachers. Teachers realized ‘He’s [the union presi- 
dent] right, they didn’t fire everybody.” The union president concluded, “there is 
much more acceptance [of the evaluation program] now.” 
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In short, the increased identification of teachers as highly effective educators, the 
streamlined pathway out of the classroom for underperforming teachers, and the 
focus on instructional practice throughout the district has led the program to earn 
the support of a range of stakeholders. 


Facilitators 

Probably the most influential facilitator of the teacher evaluation program’s 
continued acceptance and success has been the involvement of teachers and their 
union in the reform’s initial design and ongoing implementation. Many teachers 
said that the involvement of teachers and the union helped them have faith in the 
reform. One high school science teacher said that she “liked the idea that it was 
put together by the union, the teachers, the district — that everybody participated 
in the process.” When asked whether she received less resistance to this program 
than to the prior evaluation system, one elementary school teacher responded 
with a resounding “Yes.” She also said that when teachers expressed nervousness 
about the new evaluation program she would respond: 

It’s just what we've all signed up to do and your union voted on it. Your union 
was actually probably more involved in the process than the administrators' 
union in crafting it. ... it's your document. You guys came up with it. You 
approved it. ... it wasn’t an administrator-created document like the past. 

The director of the evaluation program said that this aspect of the reform was 
especially important when challenging and potentially divisive decisions were 
being made. “It’s made a huge difference to be able to tell teachers, ‘This is what 
you as a group decided you want.’ When we have those difficult discussions it’s 
really important.” 

More broadly, the ongoing relationship between the teachers union and district 
leadership has facilitated the evaluation program’s success. Several district and 
union leaders noted that stakeholders had begun the process of developing the 
teacher evaluation program by drafting a set of shared understandings. The pro- 
gram’s director recounted that it “started from a common set of beliefs and using 
them as a touchstone. When it gets heated, we come back to agreements. That’s 
helped.” She added, “We make a conscious effort to keep collaborating. It’s more 
time-consuming. It’s easier to just make decisions sometimes, but it’s important to 
have the conversation first.” 
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Both the president of the teachers union and the director of the evaluation 
program reported that there have been no grievances with the evaluation process 
filed to date. “We try to work it out first with the administrator,” said the presi- 
dent of the teachers union. If this doesn’t solve the problem, the issue goes to the 
union president and teacher the director of the evaluation program. The president 
concluded, “Most issues are resolved at the building level” rather than through the 
district’s grievance procedure. 

Representatives of both labor and management pointed out that discussions could 
at times get contentious, and sometimes both sides needed to suspend discussion. 
“We collaborate but we don’t always agree,” explained the union president. “It’s 
not all sweetness and light,” added the assistant superintendent. Critically, both 
sides were able to return to the initial guiding principles, and discussions usually 
commenced again within days. 

Key to this ongoing collaborative relationship was the willingness to compro- 
mise on both sides. Even though, as the teacher evaluation director said, “there’s 
not always complete happiness” between the district and union, “without [the 
union’s] support we would not be moving on [dismissing] teachers.” The union’s 
willingness to support the district’s efforts to remove ineffective teachers has 
meant that the district has had to be especially diligent about holding school 
administrators accountable for carrying out the evaluation program fairly and 
with fidelity. The program’s director said that she has been forced to have “tough 
conversations with principals” about sticking to component weights and rating 
a teacher lower than his or her student-performance indicators and classroom 
instruction would indicate. 

The district has also taken extra steps to make sure that the evaluation process is 
fair and transparent. As the director of the evaluation program explained: 

I think we’ve built in mechanisms to ensure that it's fair. I think the validation 
goes a long way with that. I also think that when we looked at those hard cases 
at the end of the year, we almost bend over backwards to ensure that the process 
has been followed with fidelity [and] if there's any question about it not being 
fair that we err on the side of the teacher. 
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The current state of the teacher evaluation 
program and areas for growth 


Though most study participants queried viewed the district’s teacher evaluation 
program positively many suggested concrete ways in which the system could be 
improved. “I think we’ve made a lot of progress. There are a lot of positive out- 
comes/’ said the evaluation program’s director. She said, however, that “there’s still 
a lot of work we’ve got to do.” Many participants said that there needed to be more 
standardization of the process and the ratings among administrators. The director, 
for example, named “firming up the process” as a key area of improvement. By this 
she meant standardizing the process across all administrators. One way to accom- 
plish that is to focus on calibration in observation and student-learning objective 
goal setting across administrators, which is seen as a key area of weakness, accord- 
ing to both district leaders and teachers. “We need to get better at that. Sometimes 
goals [student-learning objectives] are too high or too low,” said the program’s 
director. She added that having more consistency across evaluators will help to 
facilitate even greater acceptance of the evaluation ratings. 

Another area of concern noted by district leaders involved the possibility that 
there might have been some inflation of ratings in the first two years of the evalu- 
ation program. Speaking about the number of teachers in the 4 (strong) category 
on summative ratings, the program’s director said, “It’s great if there actually are 
so many [strong teachers]. But it maybe skewed upwards. Maybe goals aren’t as 
rigorous as they could be. It doesn’t jibe with what we see in student achievement.” 
Continuing, she said, “We expected to see more teachers in the ‘developing’ 
category given the number of new teachers every year. I think we’re going to see, 
as administrators get better calibrated in instructional practice and goals, maybe a 
downward correction.” Creating the needed rating balance will also involve reset- 
ting teachers’ expectations, acknowledging that being rated as “developing” is not 
a bad thing, particularly for early-career teachers. 

Additionally, the district is working to provide administrators with more time to 
observe teachers and discuss instruction. As the union president said, this involves 
rethinking assumptions about time use. “Do you really need a $125,000 adminis- 
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trator to do bus duty?” he asked. Furthermore, the evaluation program’s director 
has worked to streamline the evaluation process to make the reporting burden less 
cumbersome. The director noted that the new online data management system 
has made it easier for evaluators to report data and for her to track the process at 
the teacher, administrator, or school level. This also allows her to provide more 
support and accountability for school administrators. 

Finally, the district is attempting to shift away from the practice of labeling teachers 
using a numbers system to a rating system that assigns teachers performance labels. 
“Psychologically, [a numerical label] can be very damaging,” said one union stew- 
ard. Hearing this complaint, the district is making an effort to move to a qualitative 
labeling model using terms such as “effective” and “exemplary” instead of numbers. 
Although the district has always had qualitative labels to accompany numerical 
ratings, administrators and teachers have tended to use numerical ratings in prac- 
tice. The district is now making a more concerted effort to use only the labels in all 
presentations and discussions about the program within and outside the district. 


36 Center for American Progress | Reforming Teacher Evaluation: One District's Story 


Conclusion 


With the Studyville School District’s evaluation program well into its third year 
this fall, the question is: What lies ahead? 

Participants in our study were optimistic about the program’s continued existence. 
The president of the teachers union said that the evaluation program’s relatively 
smooth first two years were a good sign for its continuation. “I was pretty confi- 
dent that things would go well, but who really knew because [it was a] brand new 
system in a school district that’s not huge but not small. There [were] capacity 
issues. We have money problems like everybody else. It’s kind of surprising that it 
did go so well.” 

When asked whether he thought the evaluation program would survive the next 
contract negotiation set to begin the following school year (2012-13), one union 
steward said that he thought the program would remain in place. “Now that it’s 
there, it’s almost going to be impossible to get out. And we’ve been on national 
television and our superintendent and our union president have been down to 
Washington to talk about the successes of educational reform.” He added, “The 
general public is out here saying, ‘Schoolteachers all make too much money. 
Schoolteachers only work half a year. Schoolteachers this, school that.’ It is at least 
some way to hold teachers accountable, so I don’t see it going away.” Moreover, 
study participants generally felt positive about the reform, and as such most 
thought it should continue to exist in the district. 

In sum, our findings indicate the following: 

* Economic, political, and policy factors facilitated the development and 
success of the teacher evaluation program. These factors played a large role 
in our study’s school district in creating the will of key stakeholders to adopt 
teacher evaluation reform and invest in it over time. These factors help explain 
the reform’s favorable review from teachers, school-based administrators, and 
district and union leadership. 
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* Collaboration was at the heart of the teacher evaluation program's creation 
and development. The teacher evaluation program in the Studyville School 
District was conceived through the district’s 2009 collective bargaining agree- 
ment with its 1,600-member teachers union. Teachers and school adminis- 
trators have worked together to develop many of the key components of the 
reform. Both district and union leaders created the framework through which 
the program was created, and they continue to oversee it in a generally collab- 
orative manner. The program therefore represents collaboration between both 
teachers and administrators and the union and district leadership. It is worth 
noting that the one district in the country where teacher evaluation has been 
shown to improve student achievement, Cincinnati, also initiated its teacher 
evaluation system through a collective bargaining agreement. 14 Moreover, like 
Studyville’s teachers union, the Cincinnati Federation of Teachers has been inte- 
gral to the development and implementation of its teacher evaluation system. 

* The teacher evaluation program represents both a process and a product. 

Both the process and product of the evaluation system are aspects of the reform 
that rely on each other, and they are both important factors in the reform’s 
progress to date. The program was developed collaboratively and over time. 

The time spent developing the program in joint labor-management committees 
cultivated trust and understanding among parties. Many contingencies were also 
worked out in these meetings, and the product of the reform reflects different 
participants’ viewpoints. Participants argued that this process has been integral 
to stakeholders’ support for the program. 

* The teacher evaluation program's progress reflects strong leadership 
coupled with broad input. Leaders at the city and district level were willing to 
lead, to push the envelope, and to compromise. But they did not design the pro- 
gram alone and impose it on their constituents. They relied instead on teacher and 
administrator working groups to develop many of the details of the program. The 
program was therefore framed to address the priorities of district and union lead- 
ership. The specifics, however, reflected both teachers’ and administrators’ needs. 

To date, Studyville’s teacher evaluation program has garnered support from key 
stakeholders: teachers, school and district leaders, the teachers union, and the 
city school board. The reform has led to consequences, with some teachers being 
recognized as exemplary and others leaving the district for performance reasons. 
There is some indication that teachers feel the reform focuses them on student 
performance. 15 While the program’s direct impact on teachers’ instruction or 
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student achievement has not yet been examined, stakeholders’ favorable views of 
the program suggest that the reform maybe gaining the traction in the district that 
would allow it to affect these key outcomes. So what can other school districts can 
learn from the experience of teacher evaluation reform in our study district? 

• Get out in front of a wave of reform, but be authentic to district needs. 

Studyville has made strides with its teacher evaluation program by responding 
early to calls to improve teacher evaluation. This has allowed Studyville to stake 
out its ground in this area of reform. Studyville’s status as an early implementer 
and its collaborative stance to evaluation reform led to recognition from promi- 
nent government and labor leaders, which likely reinforced its value to key local 
stakeholders and encouraged them to continue to search for productive compro- 
mise. Moreover, by getting out in front of teacher evaluation reform, Studyville 
was able to set its own course for change and design its own system. To some 
extent, then, it was able to influence state policy rather than be confined by it. 

Obviously, however, not all districts can get out in front of reforms. What can 
districts that engage in the process of reform later than Studyville learn from 
its example? Even those who are not the first to engage in change can seize the 
opportunity to design creative solutions to key problems. When leaders see a wave 
of reform building, then can use this pressure to convene stakeholders to devise 
productive plans for moving their system forward. 

• Invest in collaboration but know its challenges. Collaboration played a major 
role in the progress of this reform. But one should not conclude that collabora- 
tion, in and of itself, is the key ingredient to a reform’s progress based on this 
study alone. In the case of our study district, the context shaped collaboration 
among the parties, encouraging them to compromise and find mutually ben- 
eficial solutions. The lesson here is that investing in collaboration can lead to a 
better outcome for all, but that collaboration is hard earned and dependent on 
context and people. Both management and labor leaders should take stock of 
these factors when seeking to engage in collaborative reform. 

• Pay attention to process and product. The teacher evaluation program’s prog- 
ress to date is as much due to the process through which it was developed as it is 
to the structural elements that comprise the reform. The time spent developing 
the program has been critical to its gaining traction in the district. Specifically, 
the time spent by teachers and administrators and district and labor leaders 

in designing the program helped to develop trust and understanding among 
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the parties. It also facilitated one of the main goals of the reform: to develop a 
common vision of effective teaching. The more time teachers and school leaders 
discuss this vision in a structured way, the more likely it is to affect instruction 
within and outside of the boundaries of the program. 

To truly improve teacher evaluation, changes must occur not only in the struc- 
ture of teacher evaluation but also in the culture that surrounds and supports 
it. Teachers must become more comfortable having others, whether they are 
administrators or teachers, observing their instruction and offering constructive 
feedback on how to improve. Attention to the process of developing a new teacher 
evaluation system may have greater success in facilitating these cultural changes 
than an exclusive focus on crafting the structural elements of an evaluation system. 

On the whole, our study participants reported that the first two years of the 
teacher evaluation program in the Studyville School District have been challeng- 
ing but productive. In the challenges and successes of Studyville we see opportu- 
nities for what teacher evaluation might become, with teachers becoming more 
skilled through the process and students benefiting in the end. 
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